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ABSTRACT 

In order to ascertain what data elements should 
commonly be considered in the development of a meaningful procedure 
for recording and analyzing library reference .transaction data for 
planning purposes, a comparison was made between the development, 
use f and results of surveys designed and tested at the University of 
Nebraska at Omaha (UNO) and the University of Illinois at 
Urban a-Champaign (UIUC) . It was concluded that although the 
computer-assisted approach utilized at UNO allows for greater 
efficiency in data analysis, the UIUC manual approach can provide 
similar results at considerably less cost in libraries without access 
to data processing equipment. In addition, the UIUC survey form was 
found to be easy to modify in order to accommodate analysis of 
reference activity in general or specific aspects of subjects 
depending on the need and the environment where the survey is 
conducted. A comparative overview of the two survey methods is 
presented in this report. Appendices comprise the UIUC survey form 
and three worksheets for UIUC data analysis. (Author/ESR) 
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INTRODUCTION 

Since there are currently no standards to adequately measure and evaluate 

reference service effectiveness, each library must act independently when designing 

a descriptive reference statistics procedure. Although the research on this subject 

indicates libraries are commonly interested in the accumulation of reference 

statistics to facilitate planning and improve upon the delivery of reference 

services, the absence of standards has produced inconsistencies in the particular 

methodologies employed for this purpose. There are continuing efforts to overcome 

this problem through the establishment of a standardized format for the collection 

and reporting of descriptive reference statistics. However, agreement on the overall 

design, content and method of implementation of these standards has not been reached. 1 

Consequently, until comprehensive, mutually acceptable standards to measure reference 

service are commonly applied, it is likely that evaluative studies in this area will 

.2 

continue to be exploratory, inconsistent, and indeterminate. 

This situation has placed some limitations on the proliferation of concerted 
studies in this area. However, interest in designing studies to investigate and 
improve upon various aspects of reference service within individual libraries has 
continued. Although the purpose and, subsequently, the methodologies of these 
studies may vary from library to library, of the known studies, there is evidence 
that a pattern is developing regarding certain data elements which appear to be 
commonly surveyed and are generally considered important when implementing a reference 



This paper concentrates on the feasibility and reliability of uniformly 
testing commonly surveyed data elements at different libraries within the context 

e#--a-rp£ej^>-nc&-sAati sties procedur e designed to as sist planning. The under- 

lying assumption is, if these common data elements were to be incorporated 



statistical procedure to facilitate planning. 
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into a standard survey form and similarly tested in more than one library, perhaps 
consistent, more meaningful information could be obtained to exact a better 
understanding of the reference process. This information could provide a basis . 
for overall planning of library service activities, and might also assist in the 
development of standard measurements from which uniform reference statistical 
data could be collected and evaluated. 

In order to assess the feasibility of this concept, similar reference 
statistics procedures were designed and tested at the University of Nebraska at 
Omaha, University Library (UNO) and the University of Illinois at Urbana-Champaign, 
Undergraduate Library and Main Library Reference Department (UIUC). In the 
comparison of these surveys, the procedural advantages and disadvantages to the 
final design utilized at each institution will be assessed through the discussion 
of the development, use and results of each study. \ 
BACKGROUND-UNO STUDY 

The usefulness of recording reference statistics has been argued from various 
extremes. On the one hand, many librarians would agree that statistics are 
useful for planning purposes, however, others have observed that the collection 
of statistics contributes little to the .functioning of the 1 i bra ry program and 

* 

may, in fact, prove detrimental by deflecting the library's finite personnel 
resources in the performance of profitless tasks. 4 The compromise on this 
issue should rest somewhere in, the ability to collect as much meaningful informa- 
tion as possible, without presenting undue constraints on those collecting and 
analyzing the recorded data. In the development of the UNO and UIUC surveys, 
an attempt has been made to negotiate this compromise. 

The primary reason for the development of the UNO survey was to redirect 

the extensive r eferencTpe«6i^^ 

transaction information of limited value. At that time, the process of producing 
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reports on collected data alone took nearly 400 hours of staff time each year. 
Based on an average of 12 transactions per hour, this conceivably resulted in 
the loss of potential service to 4,800 patrons annually. Recognizing this 
deficiency, remedial alternatives were explored through investigation of other 
methods to collect more meaningful information, but with greater efficiency. 
DESIGN-UNO STUDY 

The available research on reference statistics provided much useful informa- 
tion regarding collection procedures and types of data to include in the design 
of a more appropriate procedure. After a review of the studies in this area, 
the decision to include certain data elements in rhe UNO and, ultimately, the 
UIUC survey was derived from the result^ of the 1977 survey of reference/information 
statistics forms and procedures conducted by the ALA Committee on Statistics for 
Reference Services. 5 This survey revealed, of th6se libraries utilizing 
statistical forms for collecting reference data, there was some evidence of a 
consensus on those data elements most commonly surveyed for evaluative, planning 
purposes. Those areas found most often surveyed were: day of the week, time 
of day, year and month, type of reference transaction, form of request, type 
of material used to answer questions, duration of transaction, subject area, and 
status of the answerer. These areas depicted the most comprehensive, commonly 
.analyzed aspects of reference service found from one library to another. 
Therefore, it was decided to test the usefulness of these common data elements 
in a reference statistics procedure designed to facilitate planning of overall 

reference department activity. 

Although these data elements represented the optimum coverage of what 
could be surveyed, if was apparent the manual processing of information this 
^^^eo-Vd^^iffiposp a g reat bur den on personnel collectin g and analyzing 

i 

this data. Consequently, alternatives in the method of collection and analysis 
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of the recorded data were explored, For this purpose, substantial information 
regarding various procedures for reference data collection and manipulation was 
obtained from information provided at the 1977 ALA Pre-Conference on Reference 
Statistics for Reference Services. The Pre-Conference was intended to provide 

guidance on sampling and other statistical methods that could identify areas 

fi - 

susceptible to improvement without imposing a heavy burden of observations. The 
work sessions were designed to instruct participants in various sampling 
techniques, as well as introducing methods for analyzing collected data including 
use of the Statistical Package for the Social Sciences Program (SPSS). Although 
sampling and the use of computer programs to analyze data made recording and 
manipulation of collected data more efficient, there was no discussion of a means 
to eliminate the mandatory key-punching of information from completed survey forms 
to a compatible machine-readable format. If this task could be eliminated using 
a machine- readable encoding device, data could be immediately and efficiently 
processed through a supporting computer program. Except for the use of a direct 
input terminal at the reference desk, which, altough not totally unimaginable, was 
impractical, data collection on machine-readable optical mark scan sheets provided 
the only workable alternative for the overall purpose of the UNO study. 

In order to determine if this approach was* financially feasible and compatible 
* with local data processing capabilities, information regarding cost estimates 
for developing this procedure and sources of available support was solicited from 
the UNO Office of Computer Operations. Their estimated cost for developing this 
procedure including programmers and project analyst's time, the creation of a plate 

•r " 

to print 10,000 scan sheets and operation of a supporting computer program was 
$5,700 over two years. As indicated earlier, the amount of time spent .processing 
reference transaction information with the existing manual approach 
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was 400 hours annually. This represented approximately $3,200 in personnel 
expenditure and resulted in the potential loss of service to an average of 4,800 
patrons per year. Since the proposed automated procedure proved cost effective 
and the expected service and administrative benefits were significant, the 
support for the project was successfully obtained through a grant partially 
funded by the Information Systems Executive Council of the University of Nebraska 
System with matching funds supplied by the UNO University Library. 
DATA ANALYSIS-UNO STUDY 

The practicality of utilizing optical mark scan sheets to record reference 
transaction information provided an efficient approach to realize one major 
objective of the UNO study.' Collection of data in this manner provided a means 
to survey extensive data concerning reference activity while substantially 
reducing the need. for time consuming analysis of reference transaction information. 
To expedite this, a computer program was developed to read and process information 
from scan sheets designed to reflect data on reference transaction activity in t 
the following areas: day of the week, time of day, year and month, type of 
reference transaction, type of material used to answer question, form of request, 
duration of transaction, subject area, and status of the answerer. 

Since the data collected from these areas was to assist with planning, the 
computer was programmed to produce monthly and year-to-date reports on activity 
in all these areas. The information from these reports provided the ability to 
analyze overall and specific activity of reference service based upon the day 
of the week, time of day, question type, subject, resource used, duration of 
transaction, and the answerer's level of expertise. Data manipulation and display 
in this manner was necessary to accommodate another desirable objective from the 
procedure. Detailed information regarding reference service demands could be 
monitored so appropriate staffing patterns and staff development in high demand 
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areas could be implemented. This was possible since the program processed data 
acquired on all transactions and then categorized it based on professional and 
pre-professional ancounters at the reference desk. In this respect, professional 
and pre-professional total and average duration of time with each transaction by 
subject was available. This provided the ability to learn more about reference 
st^ff capabilities and limitations with specific subjects and if, in fact, overall 
efficiency could be correlated with previous training and experience. 

In a recent article, "Designing Optical Mark Forms for Reference Statistics", 
by Clark Hallman? information is provided detailing the UNO expeience with this 
procedure for collecting and analyzing these commonly surveyed data elements for 
planning purposes. Hallman observed, the immediate advantage to this procedure 
was the prompt, labor-free tabulation of data available in usable form within days 
of the collection periods. 8 In addition, the cumulative, long-term results were 
useful for scheduling departmental activities, reference collection development 
and Ljdgeting based upon the knowledge of subjects and resources used, and staff 
development in high demand areas. The results of the UNO study provided one 
measure to support the presumption that these commonly surveyed data elements 
could be reliably and efficiently tested with this procedure and did provide 
useful information for planning reference department activities. 

Although the. UNO experience with this reference statistics procedure was 
considered positive, this particular methodology did present a number of 
undesirable restrictions and limitations. Over a period of time* it was discovered 
that a major disadvantage in using this procedure was the rigidity of the program 
itself. To alter the program to reflect new experiences of significant additional 
information observed through reference service encounters meant costly adjustments- 
in printing new optical mark scan sheets and, subsequently, the supporting computer, 
program. 

> 
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This negative aspect in the methodology of data collection posed a difficult 
problem in the development of the test of these same data elements for the 
comparative survey conducted in the U1UC libraries. As uniformity in the survey 
designs was essential to produce meaningful results, the same, commonly , surveyed 
data elements had to be similarly tested in the UIUC survey. However, in order 
to accommodate the need for greater flexibility in the design of the UIUC survey, 
it was doubtful that optical mark scan sheets could be utilized. Consequently, 
to obtain a desirable degree of flexibility in the UIUC survey, the significant 
advantages regarding the efficiency of data processing afforded through the use 
of scan. sheets would be lost. This problem was further complicated by the fact 
that the UIUC survey would be conducted at more than one location making survey 
flex-ibility even more desirable. 
DESIGN-UIUC STUDY . 

The resultant compromise was to design a comparable, yet efficient manual 
test of these same data elements which allowed for greater flexibility in both 
data collection and analysis. Although this did result in some procedural 
inconsistenci.es in the two surveys, the changes were necessary to improve the 
effectiveness of the basic survey design and enhance the likelihood that it 
could be adapted to a more diverse library environment. 

In the design of the form to record data, each area involved in the UIUC 
survey was consulted to assist in the determination of the overall and, 
specifically, the subject content to- be represented on the final form (Figure 1) 
'This was done to obtain a consensus of understanding on the design of the form 
and to ensure that unique subjects to specific reference service points would 
be surveyed and have direct applicability for planning in that area. Once 
agreement on the design of the form had been reached, specific directions 
regarding the appropriate procedure for completing the forms were presented to 
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each'area involved in the survey. The survey was then implemented -oVer a 
randomly selected period to represent three weeks of the Spring Semester of 1981. 

Within the survey form (Figure 1), each type of activity was to be noted by 
simply checking the appropriate category. Completing a form in this manner was 
no more burdensome than the process of blackening the appropriate circles as 
required with the optical mark scan sheets. In fact, this method was somewhat 
faster as the person collecting the data did not have to be so concerned with 
exactly filling in ttr circles to ensure their machine readability. In addition, 
all of the data was presented on the survey form and reference to separate tables 
of subject categories to complete the corresponding numeric code as required on 
the scan sheets was eliminated. Many of the categories for recording reference 
activity on the form are self explanatory. However, some information regarding 
the approach for completing the sections is warranted. 

When designing the form to record data concerning QUESTION TYPE (Figure 1), 
an attempt was made to develop a survey form which could categorically induce 
the individual to conform to the definitions outlined in the Library General 
Information Survey (LIBGIS) regarding reference or directional transactions. 8 
As the analysis of the completed data depended on this basic understanding, it 
was essential that all participants in this survey clearly distinguished between 
the type of transaction they encountered. By including several possibilities 
to describe a transaction, new survey participants could more accurately complete 
the form without the need for. training sessions on each occasion. 

The DATE was to be recorded by year, month, day, e.g. 81/10/17. This could 
be modified through any number of variations but was conducted in this manner 
in the UIUC survey simply to allow for the future possibility these forms would 
be input through the SPSS program. The TIME of day was recorded on the 24-hour 
clock for the same reason However, this could easil y b e represented by 
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indicating the time and checking a.m. or p.m. designators. The MODE of the 
queition and LIBRARY where the question occurred should be self explanatory. 
The PATRON type was included to develop an awareness of the clientele served by 
each service point and was obtained by simply asking the patron to supply the / 
appropriate information. The ANSWERER category reflected the status of the person 
completing the survey form. DURATION was simply the amount of time necessary to 
complete the transaction. As indicated, the QUESTION TYPE category was designed 
to provide guidance on the type of question. However, this category was also 
delineated in this manner to provide information to analyze specific types of 
reference questions as they were negotiated through a particular subject 
designator. 

In the UNO survey, those resources used to answer a question were simply 
tallied to provide a cumulative use count and could not be connected to a subject 
within a transaction. In the UIUC study, the use of resources by type, such as 
card catalog, index, abstract, etc. was not delineated by type of resource. 
Rather, the nature of the inquiry was emphasized. This was done to provide 
data to investigate the particular strengths and weaknesses of the collection to 
accommodate various types of reference inquiries against a specific subject. 
In this respect, the ANSWERED SATISFACTORILY and ANSWERED UNSATISFACTORILY 
categories were unique, but important to the UIUC survey since it was conducted 
at more than one service point. In this category, a question was to be 
considered answered satisfactorily when the answerer believed he had provided 
the patron with the appropriate source or response to his question. A question 
was to be considered answered unsatisfactorily when the answerer could not 
satisfy the patron's information needs at the point of inquiry based upon the 
delineated reasons. Although there is considerable dissension regarding patrons' 
and librarians ' perceptions of what consti tutes a satisfactory response, the 
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analysis of data collected through the inclusion of this category in this manner 
does provide the ability to evaluate reference service effectiveness against 
several important criteria. " 

The SUBJECT area of the form was to. be completed by indicating the primary 
subject interest of the patron. Again, this was to be determined by the person 
answering the question. Since only one subject could be checked, the survey 
relied heavily on the individual's completing the form to ascertain the appropriate 
subject from the list provided. However, if the patron's inquiry was in a 
subject not represented on the form, the "Other" designator could be marked, and 
the subject could be specified. This aspect of the UIUC survey allowed for the 
necessary flexibility to monitor new or unique subjects encountered in the 
delivery of reference service. Essentially, any list of subjects could be tested 
with this procedure as alteration of the subject designators does not affect the 
process for analysis of recorded data. This flexibility allows for general or . . 
very specific tests of subject use to be conducted depending upon the environment 
or type of library. When determining developing areas of interest for forecasting 
purposes or monitoring specific areas Of use, this flexibility could beWficial 

If the inquiry had no subject, but was not directional by definition, the 
'"No Subject-Library Related" designator was to be checked. For example, it was 
possible to perform a verification or Library Computer System (LCS) function 
thlt had no specific subject, .but Was library related. Similarly, questions 
such a*. "Where are the indexes?" or /'Do you have the Reader's Guide?" were 
checked as reference, but regarded as "No Subject-Library Related". These 
questions, do not fall into the category of a verification type defined as 
determining Existence of an author and the form of their name or the correct 
title of a particular work. However, these questions could not be regarded as 
purely directional questions as they assume a level of acquired experience on the 
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part of the answerer to recognize the information needed is library related. 
Although this is a difficult area to test effectively, the analysis of data 
acquired from the "No Subject-Library Related" designator does provide some 
measure of the time required to respond to patrons' library related questions. 

DATA ANALYSIS-UIUC STUDY 

Originally, each area was provided with an allotment of forms determined 
by multiplying the average number of questions per day, acquired from previous 
semester's statistics, by 21, the number of days in the sampling period. Once 
each area had depleted its allotment of forms, the data was transferred to a 
set of three worksheets for analysis (Figures 2, 3, & 4). 

After some experimentation with this procedure, it was discovered the 
most expedient method for analyzing recorded data by library was to first 
separate the completed forms into two groups based on Question type, discounting 
those incompleted forms. In the UIUC survey, directional questions were simply 
counted as a part of the total number of questions. However, these questions 
could be transposed to Figure 2 if more detailed analysis of this activity 
was needed.- The remaining group of reference type questions must then be 
separated by subject designator. The activity in each subject area is then 
calculated through a corresponding set of 3 worksheets. At this point, the 
development of a file or similar arrangement by subject with accompanying 
worksheets is recommended for control in record keeping. 

Once the forms have been grouped by subject, an observation from the 
survey form should be recorded on the worksheet by placing a tick in the 
appropriate square of either Figure 2 or 3. The data collected on Figure 2 
supplies a visual, as well as a numeric, display of subject activity by time of 
day, day of week, mode of inquiry, and patron type. Since the UIUC survey, was 
conducted in more than one library, the PATRON TYPE category was included to 
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provide data regarding the characteristics of the clientele served at each 
library. This information was useful for determining .the overall appropriateness 
of the collection and whether emphasis or de-emphasis in certain areas should 
be considered or a redirection of clientele through suitable outreach programs 
should be enacted. The data collected on Figure 3 provides a more detailed 
account of the disposition of the subject transaction based upon the status of 
the answerer and the time required to perform a particular function. If more 
descriptive information regarding subject activity is required, the data from 
Figure 3 can be further calculated through one or more of the conditions outlined 
in Table 4. 

The calculations in Table 4 can be performed once the total activity in each 
of the, categories for Table 3 is known. This can be weekly, monthly, yearly, or 
whatever time period is appropriate depending on the nature of the survey sample 
or the intended purpose of the accumulated^data. Table 4 is structured to 
provide the ability to assess percentage of activity against total activity 
in each possible category represented in Table 3. The information supplied from 
this table can indicate where the activity in a subject area is concentrated by 
type of reference question and the extent of time required to negotiate a ; 
satisfactory or unsatisfactory response dependent upon the status of the answerer 
It is also possible to obtain information regarding the reasons for supplying 
an unsatisfactory response in a certain subject area. This information could 
be useful for monitoring collection maintenance procedures, the type of clientele 
not served, through a particular portion of the collection, consideration of 
collection development in identified areas, and the need for possible staff 
development in certain subject areas. 
CONCLUSION 

The primary purpose for conducting these studies was to .investigate what 
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data elements should commonly be considered in the development of a meaningful 
procedure for recording and analyzing reference transaction data for planning 
purposes. In this respect, the information acquired through the use of the 
UNO or UIUC reference statistics procedure can provide relevant, reliable data 
to assist with overall planning of reference department activities. Although 
the computer-assisted approach utilized at UNO allows for greater efficiency 
in data analysis, the UIUC manual approach can provide similar results at 
considerably less cost in libraries without access to data processing equipment. 
In addition, the UIUC survey form can be easily modified to accommodate analysis 
of reference activity in general or specific aspects of subjects depending on 
the need and the environment where the survey is conducted. This flexibility 
allows for these same data elements to be incorporated and similarly tested 
in various types of libraries. 

The uniform testing of these data elements in the UNO and UIUC libraries 
provided an indication that this type of data collection and analysis could 
provide information to assist with the development of standard measures for 
further evaluation of reference service. In several areas, the concentration 
of activity experienced in .one library was paralled by a similar degree of 
activity in the other two libraries. If more libraries were to similarly test 
these data elements, the cumulative results could provide the necessary data 
to develop standard procedures /for measuring and evaluating reference service. 

There are other studies which could be conducted utilizing the data acquired 
from this type of reference statistics procedure. For example, in all three 
libraries, Business, Education, Medicine, Literature, Political Science, and 
Sociology appeared within the list of the 10 most heavily requested subject 
areas with Business and Education ranking first or second at each library. If 
these general subjects were to be isolated, tested through more specific 
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designators accompanied by careful monitoring of referrals, more detailed 
information for collection development purposes could be obtained. It might 
then be possible to use this data to determine what, if any, correlation exists 
between reference and circulation activity. In this respect, procedures to 
project activity, such as the linear regression model discussed by Harter and 
Fields 9, could be tested to determine whether any significant cause-effect 
relationship exists regarding use of library materials and services. 

There are undoubtedly other applications for the use of this data. 
Hopefully, the information acquired through use of these data elements will 
prompt such endeavors. 
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Lack 

Expertise ; ■ 

Other 



FXGUAE H 



Subjects 



Total questions: 



Pro. 

Pre-pro. 

Sat. 
Unsat. 

Pro* Sat* 
Pro* Unsat. 

Pre. Sat* 
Pre* Unsat. 




Reason Unsatisfactory: 

Source* Unavailable 

No Adequate Source Referral 

No Adequate Source in Library 

» 

Lack Expertise * 
Other 



Total Duration: 

0-1 . 
2-5 
6-15 
16-59 
60+ ; 

Ave* Pur*' win. 
Tot. Dur. ^_ hrs* 



Total Duration Professional: 

0-1 . X 

2-5 

6-15 

60+ 



Total Duration Satisfactory: Total Duration Unsatisfactory: 



Average Duration 
Total Duration 



nln. 
hrs. 



0-1 

2-5 . _ 

6-15 

60+ 

Average Duration 
Total Duration 



inin* 
hrs. 



0-1 _ 

2-5 ; _ 

6-15 ; _ 

60+ __ _ 
Average Duration 
Total Duration 



Bin. 
hrs. 



Total Duration 

Pre-professlonal: 

0-1 Z 

2-5 
6-15 

60+ __ _ 

Average Duratio n n ln 
Total Duration^ hrs 



Duration Satisfactory 
Professional: 

0-1 _X 

2-5 

6-15 

60+ : 



Average Duration 
Total Duration 



nln. 
hrs. 



Duration Unsatisfactory 
Professional: 



0-1 ■ _ 
2-5 ■ _ 

6-15 _ 

60+ _ 

Average Duration 
Total Duration 



nln. 
hrs. 



Duration Satisfactory 
Pre-professlonal : 

0-1 X 

2-5 „ 

6-15 * 

60+ . 

Average Duration ____ tr.in. 

Total Duration 



Duration Unsatisfactory 
Pre-prof essior.al: 

0-1 ■ X 

2-5 

6-15 

60+ 

Average Duration^^^^ jaln 
Total Duration hrs 
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